There are two parts to getting Processing to work with Twitter: authenticating an application and downloading the Twitter4j library. Authentication was simpler than I thought it would be. I just needed to go to the Twitter dev site, create an application, and copy down the credentials. Installing the library was frustrating. The folders and files inside the zip file I downloaded weren't named so that Processing would recognize it. After working out those two parts, I just copied Jer's code into a sketch. It produces a visualization of random words from tweets containing the hashtag #ows.
I ran into some trouble with the sketch early on. The day after I played with the sketch, I loaded it and tried it again, only for Processing to highlight one of the lines and display an error message that read "cannot convert Twitter to Twitter". A Google search for the phrase turned up fruitless. After a bit of messing around, I discovered that the code worked fine in a new sketch, but not after I saved it. I figured out that because I had named the sketch "Twitter" when I saved it, it messed up the sketch somehow.
After working out those issues, I messed around with the code for a bit to figure out how it works. Processing sends a query to Twitter with a search string and the number of tweets it wants back. It splits up each tweet into three parts: the message, the user that sent it, and the date it was sent. It breaks the message into individual words and puts them into an array. When the visualization renders, it pulls the words from this array and randomly puts them on screen.
I experimented with the sketch to figure out how to render entire tweets instead of individual words. I also didn't want them to fade out. I changed the search term to #mw3 and adjusted the size of the window to fit entire tweets. It ended up an unreadable mess, but at least I made some progress.
I eventually worked out how to create something like Shuu.sh. Shuu.sh is a web-based Twitter reader that changes tweet sizes based on how often the user tweets. If the user tweets infrequently, the tweets appear big. Tweets from talkative users appear tiny.
My sketch uses tweet length to determine how bright the tweet appears. Short tweets appear bright, while long tweets appear dark. I also tried increasing the text size according to length, but I haven't figured out the proper values to get that working without text overlap. This sketch doesn't animate, but a little more code would make it look more interesting.
Fetching live data from Twitter isn't as difficult as I thought. I'll play around with Processing to see if I can generate something interesting.
HashMaps and Bar Graphs
Lab report 2
Much of my experience with Tweet4J, the library I'm using with Processing to access the Twitter API, has consisted of digging through confusing Javadoc documentation and copying/pasting phrases from it to see what works.
What I wanted to do was to store how often users tweeted within the past 200 tweets of my home timeline. The way that I'm used to storing lists of data is in an Array. An Array is a collection of objects, with each piece of data associated with a numeric index. For example, a list of phone numbers might be found in an array like this:
I can't use an Array for counting how often someone tweeted though. Instead I used a HashMap, which works a little differently than an Array. Instead of indexing items with ascending numbers, it uses text strings. In this case, I used Twitter handles as the key to reference the number of times the user tweeted in the past 200 tweets.
The issue I had with this method is that I didn't know how to access the data once I stored it in the HashMap. With an Array, I know that each item has a number associated with it, so I can call them up with a loop of ascending numbers. This doesn't work with a HashMap, particularly one with live data as keys. To access the data in the HashMap, the sketch needs an Iterator. Despite that an Iterator appears on the HashMap page, Processing documentation doesn't explain how to use it. I ended up finding an example that I modified to work in my sketch.
I worked out a way to draw circles based on how many times a user tweeted in the latest 200 tweets of my home timeline. In this example, I randomized the circles' positions and colours.
If I constrain the colours, and allow the sketch to continue drawing after a single loop, it looks like this. This isn't very helpful for visualization, but it's fun to watch animated.
After that, I changed the sketch so that it would draw rectangles that would always be 20 pixels tall and would always start from the same x-coordinate.
I figured out how to space them evenly using a variable that increases each time it draws a rectangle.
After a few iterations, the sketch could display names and draw lines in between each bar. In most cases, this results in a list that's too long to fit on screen. I decided that since users with a single tweet outnumbered the rest, I could put them into a separate area.
Later on, I decided to ditch the single tweet users. The visualization is meant to display who's tweeting the most. Having a large number of single tweets just clutters the display. This still sometimes results in a list too long to fit on screen, so I might have to bump the threshold up. Also, instead of random colours, I chose a specific colour for the bars. The number of tweets affects the opacity of the bar, resulting in darker colours for users with fewer tweets. I also added a number of tweets beside each bar. (I still need to add a title, though.)
HashMaps don't let you order the data, which means I can't arrange users by number of tweets. I tried to find a way to fake it (i.e. only draw users with 20 tweets, then 19 tweets, etc.), but with the way the Iterator works, I can't do that. For now, the data shows up in a random order.
My next step is to make the visualization more interesting. The sketch I have right now draws the chart, then stops. I want to make it fun, interesting, and perhaps interactive.
Processing + Twitter
Lab Report 1On October 27, 2011, Jer Thorpe posted an updated version of his tutorial explaining how to access the Twitter API through Processing. Because I'm interested in using Processing and visualizing Twitter data for the final project, I checked it out.
There are two parts to getting Processing to work with Twitter: authenticating an application and downloading the Twitter4j library. Authentication was simpler than I thought it would be. I just needed to go to the Twitter dev site, create an application, and copy down the credentials. Installing the library was frustrating. The folders and files inside the zip file I downloaded weren't named so that Processing would recognize it. After working out those two parts, I just copied Jer's code into a sketch. It produces a visualization of random words from tweets containing the hashtag #ows.
I ran into some trouble with the sketch early on. The day after I played with the sketch, I loaded it and tried it again, only for Processing to highlight one of the lines and display an error message that read "cannot convert Twitter to Twitter". A Google search for the phrase turned up fruitless. After a bit of messing around, I discovered that the code worked fine in a new sketch, but not after I saved it. I figured out that because I had named the sketch "Twitter" when I saved it, it messed up the sketch somehow.
After working out those issues, I messed around with the code for a bit to figure out how it works. Processing sends a query to Twitter with a search string and the number of tweets it wants back. It splits up each tweet into three parts: the message, the user that sent it, and the date it was sent. It breaks the message into individual words and puts them into an array. When the visualization renders, it pulls the words from this array and randomly puts them on screen.
I experimented with the sketch to figure out how to render entire tweets instead of individual words. I also didn't want them to fade out. I changed the search term to #mw3 and adjusted the size of the window to fit entire tweets. It ended up an unreadable mess, but at least I made some progress.
I eventually worked out how to create something like Shuu.sh. Shuu.sh is a web-based Twitter reader that changes tweet sizes based on how often the user tweets. If the user tweets infrequently, the tweets appear big. Tweets from talkative users appear tiny.
My sketch uses tweet length to determine how bright the tweet appears. Short tweets appear bright, while long tweets appear dark. I also tried increasing the text size according to length, but I haven't figured out the proper values to get that working without text overlap. This sketch doesn't animate, but a little more code would make it look more interesting.
Fetching live data from Twitter isn't as difficult as I thought. I'll play around with Processing to see if I can generate something interesting.
HashMaps and Bar Graphs
Lab report 2Much of my experience with Tweet4J, the library I'm using with Processing to access the Twitter API, has consisted of digging through confusing Javadoc documentation and copying/pasting phrases from it to see what works.
What I wanted to do was to store how often users tweeted within the past 200 tweets of my home timeline. The way that I'm used to storing lists of data is in an Array. An Array is a collection of objects, with each piece of data associated with a numeric index. For example, a list of phone numbers might be found in an array like this:
phoneNumbers[0] = 4165550983
phoneNumbers[1] = 6475551254
phoneNumbers[2] = 9055552935
I can't use an Array for counting how often someone tweeted though. Instead I used a HashMap, which works a little differently than an Array. Instead of indexing items with ascending numbers, it uses text strings. In this case, I used Twitter handles as the key to reference the number of times the user tweeted in the past 200 tweets.
The issue I had with this method is that I didn't know how to access the data once I stored it in the HashMap. With an Array, I know that each item has a number associated with it, so I can call them up with a loop of ascending numbers. This doesn't work with a HashMap, particularly one with live data as keys. To access the data in the HashMap, the sketch needs an Iterator. Despite that an Iterator appears on the HashMap page, Processing documentation doesn't explain how to use it. I ended up finding an example that I modified to work in my sketch.
I worked out a way to draw circles based on how many times a user tweeted in the latest 200 tweets of my home timeline. In this example, I randomized the circles' positions and colours.
If I constrain the colours, and allow the sketch to continue drawing after a single loop, it looks like this. This isn't very helpful for visualization, but it's fun to watch animated.
After that, I changed the sketch so that it would draw rectangles that would always be 20 pixels tall and would always start from the same x-coordinate.
I figured out how to space them evenly using a variable that increases each time it draws a rectangle.
After a few iterations, the sketch could display names and draw lines in between each bar. In most cases, this results in a list that's too long to fit on screen. I decided that since users with a single tweet outnumbered the rest, I could put them into a separate area.
Later on, I decided to ditch the single tweet users. The visualization is meant to display who's tweeting the most. Having a large number of single tweets just clutters the display. This still sometimes results in a list too long to fit on screen, so I might have to bump the threshold up. Also, instead of random colours, I chose a specific colour for the bars. The number of tweets affects the opacity of the bar, resulting in darker colours for users with fewer tweets. I also added a number of tweets beside each bar. (I still need to add a title, though.)
HashMaps don't let you order the data, which means I can't arrange users by number of tweets. I tried to find a way to fake it (i.e. only draw users with 20 tweets, then 19 tweets, etc.), but with the way the Iterator works, I can't do that. For now, the data shows up in a random order.
My next step is to make the visualization more interesting. The sketch I have right now draws the chart, then stops. I want to make it fun, interesting, and perhaps interactive.